Utility of Word Embeddings in Language Identification

1. Overview

How it started

I was going through qdrant's UI after loading a dataset comprised of quranic text when I noticed something interesting, the datapoints can be distinguished by language through eye-ball inspection of a 2-D plot. qdrant's UI allows the querying of datapoints by their attributes and displaying the queried data in a 2-D plot obtained through dimensionality reduction techniques configurable by the user. The pattern was noticeable for both word representations produced using a commercially available LLM and publicly available LLM which led to the following questions:

What’s the utility of the word representations in language identification?
Can the same level of accuracy in language identification be achieved using a heuristic?

Key Findings - A heuristic approach that uses regular expressions to detect languages based on the script used in the text obtains comparable (or better) recall and precision on language detection, to word vectors produced using nomic-embed-text-latest (nomic v1.5), nomic-embed-text-v2-moe:latest, and gemini-embedding-001. The results for nomic-embed-text-latest is especially surprising given that it is not multilingual and was developed primarily for English (en). The text used for the assessment uses script primarily associated with the language e.g. Ge’ez for Amharic, Arabic script for Arabic (ar) and this is a key factor in the outcome of the assessment. Although cross-validation on a reasonable sample (40% of entire data) has been used to ascertain consistency in results, a larger sample may reveal a different pattern. The choice of sample size was mainly dictated by memory constraints.

So What - In situations where language is primarily expressed in the script mainly associated with it, examining the characters in the text could be more sufficient than LLM based word representations in detecting the language in the text.
- For natural language tasks such as language identification non-commercial embedding models such as nomic-embed-text-v2-moe:latest could be more suitable than commercially available embedding models such as gemini-embedding-001, given how much better it performed on this task: nomic-embed-text-v2-moe:latest outperformed gemini-embedding-001 on every metric in this assessment.

Code

import warnings
from numba.core.errors import NumbaWarning
warnings.filterwarnings("ignore", category=NumbaWarning)

from vectoriser import get_qdrant
from query_qdrant import (load_sample, load_sample_where, 
                    get_payload_keys, get_payload, 
                    sample_records, extract_payload, 
                    fetch_records_for_dataframe)

from sklearn.cluster import KMeans
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import LabelEncoder
from sklearn.decomposition import PCA
from sklearn.metrics import normalized_mutual_info_score, recall_score, precision_score
from sklearn.pipeline import Pipeline

import numpy as np
from itertools import combinations
import pandas as pd
import umap
from matplotlib import pyplot as plt
import seaborn as sb
from umap import UMAP
import umap.plot
import matplotlib.ticker as ticker
import regex
from lang_util import detect_language, language_map
from pandas.io.formats.style import Styler

1.2 Approach

To address the two questions (above), I adopted the following approach:

Word embeddings and their corresponding text and attributes were sampled from three collections on qdrant where each collection corresponds to vectors produced using either gemini-embedding-001, nomic-embed-text-latest, or nomic-embed-text-v2-moe:latest. The text consists of quranic script in a variety of languages obtained from tanzil.net. Details of the text data and the LLMs is provided in Section 2.0.
To enable like-for-like comparison between the vectors, samples were first collected from one collection and the text from the collection was used to obtain a matching sample for each of the other collections i.e. the only thing that differs between the three collections is their word vectors (word embeddings).
For each sample from the collections, a UMAP model was fitted on a subset of the sample i.e. the training set, obtaining a projection of the data on a 2-dimensional Riemannian manifold. PCA was first used to reduce the dimensions of the vectors before fitting the UMAP model. Subsequently a knn-classifier is fitted on the projected data, assigning a label (predicted language) to each record. This is similar to the BERTopic approach.
The performance of each umap+knn-classifier in language identification was compared to a heuristic approach that uses the count of characters in the text associated with specific languages for language identification. The heuristic uses regular expressions in counting the characters and it is described in detail in Section 6.1.

flowchart LR

A[word embeddings] -->|sampling| B[sample]
B -->|UMAP - Dimensionality Reduction| D[Projections]
D --> |KNN Classification| E[Language Classification]
B -->|Heuristic Language Classification|F[Language Classification]
F-->|Metrics Calculation|G[Results Comparison]
E-->|Metrics Calculation|G

2.0 Data

The vectorised quranic text obtained from tanzil.net is stored in the following collections on a qdrant instance running locally on a homelab cluster. Details of each collection and the vectorisers used is provided below:

Collection (word embeddings)	Number of Records	Vector Dimensions	Vectoriser	Source	Vectoriser No. Parameters (Millions)	Model Type
`quran-embedding-prefect-gemini-embedding-001`	249,442	768	`gemini-embedding-001`	Google	Undisclosed	Transformer (Gemini-initialised, decoder-based)
`quran-embedding-prefect-nomic-embed-text-latest`	249,442	768	`nomic-embed-text:latest`	Ollama (Local Instance)	137	BERT
`quran-embedding-prefect-nomic-embed-text-v2-moe-latest`	249,442	768	`nomic-embed-text-v2-moe:latest`	Ollama (Local Instance)	475.29	BERT - MoE

A list of the languages covered in this study and whether or not they are supported by either of the embedding models is provided below

Code	Language	gemini-embedding-001	nomic v1.5 (nomic-embed-text-latest)	nomic-embed-text-v2-moe
am	Amharic	✓	✗	✓
ar	Arabic	✓	✗	✓
bs	Bosnian	✓	✗	✓
dv	Divehi	✓	✗	✗
en	English	✓	✓	✓
es	Spanish	✓	✗	✓
fa	Persian	✓	✗	✓
ha	Hausa	✓	✗	✓
nl	Dutch	✓	✗	✓
ru	Russian	✓	✗	✓
sd	Sindhi	✓	✗	✗
so	Somali	✓	✗	✓
sq	Albanian	✓	✗	✓
ta	Tamil	✓	✗	✓
tg	Tajik	✓	✗	✓
th	Thai	✓	✗	✓
tr	Turkish	✓	✗	✓
ur	Urdu	✓	✗	✓
uz	Uzbek	✓	✗	✓
zh	Chinese	✓	✗	✓

2.1 Sampling

Samples were collected from the collections starting with one collection first and then using the text in the first collection to sample records matching the same text from the other two collections.
The data is largely comprised of text in English (en) therefore some downsampling was applied to prevent skewing the results.

Code

def downsample_majority(df,samp_col="language"):
    props = df.groupby([samp_col]).size().to_dict()
    sorted_props = sorted(props, key=lambda e:props[e],reverse=True)
    maj = sorted_props[0]
    avg_minority = int(np.ceil(np.mean([props[e] for e in props if e!=maj])))
    A = df[df[samp_col].eq(maj)].sample(avg_minority)
    B = df[~df[samp_col].eq(maj)]
    C = pd.concat([A,B]).reset_index(drop=True)

    return C

Code

collections = {"gemini-embedding-001":'quran-embedding-prefect-gemini-embedding-001',
               "nomic-embed-text:latest":'quran-embedding-prefect-nomic-embed-text-latest', 
                "nomic-embed-text-v2-moe:latest":'quran-embedding-prefect-nomic-embed-text-v2-moe-latest'}

Code

sample_prop = 0.40

Code

init = "gemini-embedding-001"
vectors_main, main_df = sample_records(collections[init],frac=sample_prop,exclude=[])
main_df['language_description'] = main_df['language'].map(language_map)

Code

freq = main_df.groupby(['language','language_description'],dropna=False).size()
freq = pd.DataFrame(freq).rename(columns={0:"Number of Occurrences"})
freq

		Number of Occurrences
language	language_description
	NaN	2527
am	Amharic	2490
ar	Arabic	4920
bs	Bosnian	2472
dv	Divehi	2493
en	English	42337
es	Spanish	2538
fa	Persian	2458
ha	Hausa	2457
nl	Dutch	2489
ru	Russian	2478
sd	Sindhi	2521
so	Somali	2547
sq	Albanian	2527
ta	Tamil	2519
tg	Tajik	2517
th	Thai	2592
tr	Turkish	2469
ur	Urdu	4909
uz	Uzbek	2482
zh	Chinese	5035

Code

samples = {}
samples[init] = downsample_majority(main_df)

Code

vectors = {}
vectors[init] = vectors_main[main_df['id'].isin(samples[init]['id']),:]

Code

samples[init].groupby(['language']).size()
%store init

Stored 'init' (str)

An example of attributes of vectors sampled from the collection corresponding to gemini-embedding-001 is shown below. The column text in the table is what the vectors were generated from.

Code

samples[init].head()

	index	id	language	name	chapter	verse	text	surah_name	revelation_order	revelation_type	revelation_note	language_description
0	203498	d10bf675-917c-590b-a5b1-579a286546df	en	hilali	12	62	And [Yusuf (Joseph)] told his servants to put ...	Yusuf	53	Meccan	Except 1, 2, 3, 7, from Medina	English
1	52727	36202e3a-29fd-585e-a4fc-0be6a48eb049	en	itani	2	223	Your women are cultivation for you; so approac...	Al-Baqara	87	Medinan	Except 281 from Mina at the time of the Last Hajj	English
2	54352	37c344fb-1a90-5da1-8389-8604c427da82	en	arberry	78	2	Of the mighty tiding	An-Naba	80	Meccan	NaN	English
3	86554	58ac7315-6d1a-5296-aa16-649a3faa1149	en	arberry	5	70	And We took compact with the Children of Israe...	Al-Maaida	112	Medinan	Except 3, revealed at Arafat on Last Hajj	English
4	25736	1a613a70-8676-5b60-8976-c3fd3b0c6dc4	en	ahmedraza	10	74	Then after him, We sent other Noble Messengers...	Yunus	51	Meccan	Except 40, 94, 95, 96, from Medina	English

A sample equivalent to that collected from quran-embedding-prefect-gemini-embedding-001 was collected from the other two collections detailed in Section 2.0

Code

filter_columns = ['language','name','chapter','verse','text']

colls  = [coll for coll in collections if coll !=init]
for c in colls:
    if collections[c]!= None:
        print("Processing {}".format(c))
        matching_records = fetch_records_for_dataframe(
            samples[init],
            collections[c],
            filter_cols = filter_columns,
            batch_size = 200,
            with_vectors = True
        )
        samp, vec = extract_payload(matching_records, exclude=[])
        samples[c] = pd.DataFrame(samp) 
        vectors[c] = vec

        assert len(samples[c]) == len(samples[init]), (
            f"{c}: expected {len(samples[init])} matching records, got {len(samples[c])} "
            "- some rows had no exact match, so the like-for-like comparison "
            "across collections no longer holds."
        )
    else:
        samples[c] = None
        vectors[c] = None

Processing nomic-embed-text:latest
Processing nomic-embed-text-v2-moe:latest

An example of attributes of sample records stored in quran-embedding-prefect-nomic-embed-text-latest is shown below

Code

samples['nomic-embed-text:latest'].head()

	id	language	name	chapter	verse	text	surah_name	revelation_order	revelation_type	revelation_note
0	d10bf675-917c-590b-a5b1-579a286546df	en	hilali	12	62	And [Yusuf (Joseph)] told his servants to put ...	Yusuf	53	Meccan	Except 1, 2, 3, 7, from Medina
1	36202e3a-29fd-585e-a4fc-0be6a48eb049	en	itani	2	223	Your women are cultivation for you; so approac...	Al-Baqara	87	Medinan	Except 281 from Mina at the time of the Last Hajj
2	37c344fb-1a90-5da1-8389-8604c427da82	en	arberry	78	2	Of the mighty tiding	An-Naba	80	Meccan	NaN
3	58ac7315-6d1a-5296-aa16-649a3faa1149	en	arberry	5	70	And We took compact with the Children of Israe...	Al-Maaida	112	Medinan	Except 3, revealed at Arafat on Last Hajj
4	1a613a70-8676-5b60-8976-c3fd3b0c6dc4	en	ahmedraza	10	74	Then after him, We sent other Noble Messengers...	Yunus	51	Meccan	Except 40, 94, 95, 96, from Medina

A comparison of the statistical signature of the sample from the collection associated with gemini-embedding-001 to collections associated with the other models is shown below. The figure shows that the samples are equivalent i.e. the text, chapter, and verse match across the three samples.

Code

fig, ax = plt.subplots(3,3,figsize=(20,20), sharex=False)
ax = ax.flatten()

for x,c in zip([0, 3, 6],collections.keys()):
    samples[c].groupby('language').size().plot.bar(ax=ax[x])
    ax[x].set_title(c)

for x,c in zip([1,4,7],collections.keys()):
    samples[c].groupby('chapter').size().plot.bar(ax=ax[x])
    ax[x].xaxis.set_major_locator(ticker.MaxNLocator(nbins=6))
    ax[x].set_title(c)

for x,c in zip([2,5,8],collections.keys()):
    samples[init].groupby('verse').size().plot.bar(ax=ax[x])
    ax[x].xaxis.set_major_locator(ticker.MaxNLocator(nbins=6))
    ax[x].set_title(init)

for a in ax: 
    a.set_ylabel("Number of Occurrences")

Code

for col in samples:
    if collections[col]!=None:
        samples[col]['language_description'] = samples[col]['language'].map(language_map)

Code

counts = {}
for c in samples:
    if collections[c] != None:
        count = samples[c].fillna('n/a').groupby(['language','language_description'],dropna=False).size()
        counts[c] = count

A comparison of the number of records in each sample is provided below. The sample is largely dominated by english scripts

Code

pd.DataFrame(counts.values(),index = counts.keys()).T

		gemini-embedding-001	nomic-embed-text:latest	nomic-embed-text-v2-moe:latest
language	language_description
	n/a	2527	2527	2527
am	Amharic	2490	2490	2490
ar	Arabic	4920	4920	4920
bs	Bosnian	2472	2472	2472
dv	Divehi	2493	2493	2493
en	English	2872	2872	2872
es	Spanish	2538	2538	2538
fa	Persian	2458	2458	2458
ha	Hausa	2457	2457	2457
nl	Dutch	2489	2489	2489
ru	Russian	2478	2478	2478
sd	Sindhi	2521	2521	2521
so	Somali	2547	2547	2547
sq	Albanian	2527	2527	2527
ta	Tamil	2519	2519	2519
tg	Tajik	2517	2517	2517
th	Thai	2592	2592	2592
tr	Turkish	2469	2469	2469
ur	Urdu	4909	4909	4909
uz	Uzbek	2482	2482	2482
zh	Chinese	5035	5035	5035

3.0 Language Identification Using Word Representations

3.1 PCA + UMAP Parameters

The 768-dimensional vectors are first reduced with PCA before UMAP is fitted on the result - this cuts UMAP’s nearest-neighbor search cost substantially, since that cost scales with the input dimensionality. Details of the parameters used for the PCA step and the subsequent UMAP fit are provided below

Code

pca_params = dict(n_components=50, random_state=1235813)

Code

umap_params = dict(n_neighbors=30,
            verbose=True,
            n_epochs=300,
            n_jobs=-1)

3.2 KNN Classifier Parameters

The parameters used for fitting the KNeighborsClassifier is provided below

Code

knn_params = dict(
    n_jobs=-1,
    n_neighbors = 20, 
    weights = 'uniform'
)

3.3 Modelling

Code

def train_test_split(df, id_col="id", train_frac=0.7):
    """ split a dataframe into train and test based on id_col """
    tr_id = df.sample(frac=train_frac, random_state=np.random.RandomState(123456))[id_col]
    tst_id = df[~df[id_col].isin(tr_id)][id_col]

    tr_flag = df.id.isin(tr_id)
    tst_flag = df.id.isin(tst_id)

    return tr_flag, tst_flag

def train_umap(vec, pca_kwargs=None, **kwargs):
    """ reduce dimensionality with PCA, then fit a umap model on the reduced vectors """
    dim_model = Pipeline([
        ('pca', PCA(**(pca_kwargs or {}))),
        ('umap', UMAP(**kwargs)),
    ])
    dim_red = dim_model.fit_transform(vec)

    return dim_red, dim_model

def fit_knn_classifier(red_vec,tr_tar,**kwargs):
    """ fit a knn classifier """ 
    k_class = KNeighborsClassifier(**kwargs)

    k_class.fit(red_vec, tr_tar)

    return k_class

def build_pred_pipeline(dim_model, cls_model):
    """ build a pipeline using dim_model (the umap model) and cls_model (the knn model)"""

    pip_line = Pipeline([('dim_model',dim_model), 
                         ('cls_model',cls_model)])

    return pip_line

Code

def compute_norm_mutual_info(df,tar_col,pred_col):
    """ Computes the mutual information based on columns tar_col (target) and pred_col (prediction) """

    tar = df[tar_col].to_numpy()
    pred = df[pred_col].to_numpy()

    mi = normalized_mutual_info_score(tar,pred)

    return mi

Code

def compute_recall(df,tar_col, pred_col,average="macro"):
    """Calculates the recall based on tar_col (target) and pred_col (prediction) """

    tar = df[tar_col].to_numpy()
    pred = df[pred_col].to_numpy()

    rec = recall_score(tar, pred,average=average,zero_division=np.nan)

    return rec

Code

def compute_precision(df, tar_col,pred_col,average="macro"):
    """ Calculates the precision based on tar_col (target) and pred_col (prediction) """

    tar = df[tar_col].to_numpy()
    pred = df[pred_col].to_numpy()

    prec = precision_score(tar,pred, average=average,zero_division=np.nan)

    return prec

Code

def style_dataframe_heatmap(
    df: pd.DataFrame, cmap: str = "YlGnBu", precision: int = 2
        ) -> Styler:
    """Returns a pandas Styler with a background heatmap pinned strictly between 0 and 1."""
    return (
        df.style.background_gradient(cmap=cmap, vmin=0.0, vmax=1.0)
        .format(f"{{:.{precision}f}}")
        .set_properties(**{"text-align": "center"})
    )

Code

def style_summary_table(
    df: pd.DataFrame,
    cmap: str = "Blues",
    vmin: float = 0.0,
    vmax: float = 1.0,
) -> Styler:
    """Applies a background gradient to all rows except the first row (e.g.

    'Number of Languages').
    """
    # First row is the count row (e.g. 'Number of Languages'); the rest are stats.
    count_row = df.index[0]
    stat_rows = df.index[1:]

    styled = (
        df.style.background_gradient(
            cmap=cmap,
            vmin=vmin,
            vmax=vmax,
            subset=pd.IndexSlice[stat_rows, :],
        )
        .format("{:.0f}", subset=pd.IndexSlice[[count_row], :])
        .format("{:.2f}", subset=pd.IndexSlice[stat_rows, :])
        .set_properties(**{"text-align": "center"})
    )

    return styled

The samples are split into a train and test set for validation purposes and a UMAP model is fitted on the portion of the sample tagged as training

Code

train_flag = {} 
test_flag = {} 
dim_red = {}
umap_model = {}

for c in collections:
    print("Processing {}".format(c))
    train_, test_ = train_test_split(samples[c])
    dim_r, umap_m = train_umap(vectors[c][train_, :],
                                 pca_kwargs=pca_params,
                                 **umap_params)
    train_flag[c] = train_ 
    test_flag[c] = test_
    dim_red[c] = dim_r 
    umap_model[c] = umap_m

Processing gemini-embedding-001
UMAP(n_epochs=300, n_neighbors=30, verbose=True)
Sat Sep 12 17:15:01 2026 Construct fuzzy simplicial set
Sat Sep 12 17:15:01 2026 Finding Nearest Neighbors
Sat Sep 12 17:15:01 2026 Building RP forest with 15 trees
Sat Sep 12 17:15:07 2026 NN descent for 15 iterations
     1  /  15
     2  /  15
     3  /  15
    Stopping threshold met -- exiting after 3 iterations
Sat Sep 12 17:15:27 2026 Finished Nearest Neighbor Search
Sat Sep 12 17:15:32 2026 Construct embedding
    completed  0  /  300 epochs
    completed  30  /  300 epochs
    completed  60  /  300 epochs
    completed  90  /  300 epochs
    completed  120  /  300 epochs
    completed  150  /  300 epochs
    completed  180  /  300 epochs
    completed  210  /  300 epochs
    completed  240  /  300 epochs
    completed  270  /  300 epochs
Sat Sep 12 17:17:08 2026 Finished embedding
Processing nomic-embed-text:latest
UMAP(n_epochs=300, n_neighbors=30, verbose=True)
Sat Sep 12 17:17:11 2026 Construct fuzzy simplicial set
Sat Sep 12 17:17:11 2026 Finding Nearest Neighbors
Sat Sep 12 17:17:11 2026 Building RP forest with 15 trees
Sat Sep 12 17:17:11 2026 NN descent for 15 iterations
     1  /  15
     2  /  15
    Stopping threshold met -- exiting after 2 iterations
Sat Sep 12 17:17:20 2026 Finished Nearest Neighbor Search
Sat Sep 12 17:17:21 2026 Construct embedding
    completed  0  /  300 epochs
    completed  30  /  300 epochs
    completed  60  /  300 epochs
    completed  90  /  300 epochs
    completed  120  /  300 epochs
    completed  150  /  300 epochs
    completed  180  /  300 epochs
    completed  210  /  300 epochs
    completed  240  /  300 epochs
    completed  270  /  300 epochs
Sat Sep 12 17:25:18 2026 Finished embedding
Processing nomic-embed-text-v2-moe:latest
UMAP(n_epochs=300, n_neighbors=30, verbose=True)
Sat Sep 12 17:25:20 2026 Construct fuzzy simplicial set
Sat Sep 12 17:25:20 2026 Finding Nearest Neighbors
Sat Sep 12 17:25:20 2026 Building RP forest with 15 trees
Sat Sep 12 17:25:20 2026 NN descent for 15 iterations
     1  /  15
     2  /  15
     3  /  15
     4  /  15
    Stopping threshold met -- exiting after 4 iterations
Sat Sep 12 17:25:32 2026 Finished Nearest Neighbor Search
Sat Sep 12 17:25:33 2026 Construct embedding
    completed  0  /  300 epochs
    completed  30  /  300 epochs
    completed  60  /  300 epochs
    completed  90  /  300 epochs
    completed  120  /  300 epochs
    completed  150  /  300 epochs
    completed  180  /  300 epochs
    completed  210  /  300 epochs
    completed  240  /  300 epochs
    completed  270  /  300 epochs
Sat Sep 12 17:27:18 2026 Finished embedding

The projected data shows distinct clusters that are comprised primarily of one main language. The colours correspond to the actual language of the text for two of the vectorisers i.e from eyeball instruction.

Code

fig, ax = plt.subplots(2,2,figsize=(15,15))
ax = ax.flatten()

for i,c in enumerate(collections):
    labels = samples[c][train_flag[c]].language
    umap.plot.points(umap_model[c].named_steps['umap'],labels=labels,ax=ax[i])
    ax[i].set_title(c)
    ax[i].set_xlabel("Dimension - 1")
    ax[i].set_ylabel("Dimension -2")

    leg = ax[i].get_legend()
    if leg != None:
        leg.set_bbox_to_anchor((1.02, 1))
        leg.set_loc("upper left")


plt.tight_layout()

Code

tar_encoder = {}
for c in collections: 
    tar_encoder[c] = LabelEncoder()
    language_values = samples[c].language.fillna("n/a")
    # detect_language (used later for heur_lang) can return '' when no candidate
    # clears MIN_CONFIDENCE, even though no ground-truth row is ever ''. Include
    # it as a known class up front so transform() never hits an unseen label.
    tar_encoder[c].fit(pd.concat([language_values, pd.Series([''])], ignore_index=True))
    target = tar_encoder[c].transform(language_values)
    samples[c]['target_idx'] = pd.Series(target)

Code

knn_clf = {}
for c in collections:
    knn_clf[c] = fit_knn_classifier(dim_red[c],samples[c].loc[train_flag[c],'target_idx'],**knn_params)

Code

lg_pred_mdl = {} 
for c in collections:
    lg_pred_mdl[c] = build_pred_pipeline(umap_model[c],knn_clf[c])

Code

train_pred = {}
test_pred = {}
for c in collections:
    tr_f = train_flag[c]
    te_f = test_flag[c]
    
    train_pred[c] = lg_pred_mdl[c].predict(vectors[c][tr_f, :])
    test_pred[c] = lg_pred_mdl[c].predict(vectors[c][te_f, :])

    samples[c]['knn_predictions'] = np.nan
    samples[c].loc[tr_f, 'knn_predictions'] = train_pred[c]
    samples[c].loc[te_f, 'knn_predictions'] = test_pred[c]
    samples[c]['knn_predictions'] = samples[c]['knn_predictions'].astype(int)

    samples[c].loc[tr_f,'Train/Test'] = "Train"
    samples[c].loc[te_f, 'Train/Test'] = "Test"

Sat Sep 12 17:27:23 2026 Building hub-based search tree
Sat Sep 12 17:27:31 2026 Forward diversification reduced edges from 1266540 to 420081
Sat Sep 12 17:27:34 2026 Reverse diversification reduced edges from 420081 to 420081
Sat Sep 12 17:27:36 2026 Degree pruning reduced edges from 481994 to 481994
Sat Sep 12 17:27:36 2026 Resorting data and graph based on tree order
Sat Sep 12 17:27:36 2026 Building and compiling search function
    completed  0  /  100 epochs
    completed  10  /  100 epochs
    completed  20  /  100 epochs
    completed  30  /  100 epochs
    completed  40  /  100 epochs
    completed  50  /  100 epochs
    completed  60  /  100 epochs
    completed  70  /  100 epochs
    completed  80  /  100 epochs
    completed  90  /  100 epochs
Sat Sep 12 17:27:48 2026 Building hub-based search tree
Sat Sep 12 17:27:48 2026 Forward diversification reduced edges from 1266540 to 424760
Sat Sep 12 17:27:48 2026 Reverse diversification reduced edges from 424760 to 424760
Sat Sep 12 17:27:49 2026 Degree pruning reduced edges from 532108 to 532083
Sat Sep 12 17:27:49 2026 Resorting data and graph based on tree order
Sat Sep 12 17:27:49 2026 Building and compiling search function
    completed  0  /  100 epochs
    completed  10  /  100 epochs
    completed  20  /  100 epochs
    completed  30  /  100 epochs
    completed  40  /  100 epochs
    completed  50  /  100 epochs
    completed  60  /  100 epochs
    completed  70  /  100 epochs
    completed  80  /  100 epochs
    completed  90  /  100 epochs
Sat Sep 12 17:28:06 2026 Building hub-based search tree
Sat Sep 12 17:28:07 2026 Forward diversification reduced edges from 1266540 to 446230
Sat Sep 12 17:28:07 2026 Reverse diversification reduced edges from 446230 to 446230
Sat Sep 12 17:28:07 2026 Degree pruning reduced edges from 538130 to 537820
Sat Sep 12 17:28:07 2026 Resorting data and graph based on tree order
Sat Sep 12 17:28:07 2026 Building and compiling search function
    completed  0  /  100 epochs
    completed  10  /  100 epochs
    completed  20  /  100 epochs
    completed  30  /  100 epochs
    completed  40  /  100 epochs
    completed  50  /  100 epochs
    completed  60  /  100 epochs
    completed  70  /  100 epochs
    completed  80  /  100 epochs
    completed  90  /  100 epochs

4.0 Results

4.1 Performance of Word Embeddings in Language Identification

The performance of the umap+knn-classification approach on all three collections was assessed using Mutual Information, Recall and Precision. nomic-embed-text:latest achieved the best performance followed by nomic-embed-text-v2-moe:latest. This is surprising particularly because nomic-embed-text:latest was developed primarily for English en.

Code

vectoriser_props = {
    "gemini-embedding-001":{"dimension":768, "n_params":"Undisclosed"}, 
    "nomic-embed-text:latest":{"dimension":768,"n_params":"137 Million"}, 
    "nomic-embed-text-v2-moe:latest":{"dimension":768, "n_params":"475.29 Million"},
}

Code

collections

{'gemini-embedding-001': 'quran-embedding-prefect-gemini-embedding-001',
 'nomic-embed-text:latest': 'quran-embedding-prefect-nomic-embed-text-latest',
 'nomic-embed-text-v2-moe:latest': 'quran-embedding-prefect-nomic-embed-text-v2-moe-latest'}

Code

group_cols = ['Collection','Vectoriser','Vectoriser Dimensions','Vectoriser Parameters','Train/Test']
mi = []
rec = []
prec = []
for c in collections:
    samples[c]['Collection'] = c 
    
    samples[c]['Vectoriser'] = collections[c]
    samples[c]['Vectoriser Dimensions'] = vectoriser_props[c]["dimension"]
    samples[c]['Vectoriser Parameters'] = vectoriser_props[c]["n_params"]

    d = samples[c].groupby(group_cols).apply(lambda df:compute_norm_mutual_info(df, 'target_idx','knn_predictions'))
    mi.append(d)

    d1 = samples[c].groupby(group_cols).apply(lambda df:compute_recall(df, 'target_idx', 'knn_predictions'))
    rec.append(d1)

    d2 = samples[c].groupby(group_cols).apply(lambda df:compute_precision(df, 'target_idx', 'knn_predictions'))
    prec.append(d2)

Code

mi_res = pd.concat(mi).unstack()
mi_res

			Train/Test	Test	Train
Collection	Vectoriser	Vectoriser Dimensions	Vectoriser Parameters
gemini-embedding-001	quran-embedding-prefect-gemini-embedding-001	768	Undisclosed	0.004197	0.029644
nomic-embed-text-v2-moe:latest	quran-embedding-prefect-nomic-embed-text-v2-moe-latest	768	475.29 Million	0.625975	0.653503
nomic-embed-text:latest	quran-embedding-prefect-nomic-embed-text-latest	768	137 Million	0.946323	0.951402

Code

rec_res = pd.concat(rec).unstack()
rec_res

			Train/Test	Test	Train
Collection	Vectoriser	Vectoriser Dimensions	Vectoriser Parameters
gemini-embedding-001	quran-embedding-prefect-gemini-embedding-001	768	Undisclosed	0.048919	0.147268
nomic-embed-text-v2-moe:latest	quran-embedding-prefect-nomic-embed-text-v2-moe-latest	768	475.29 Million	0.669763	0.707372
nomic-embed-text:latest	quran-embedding-prefect-nomic-embed-text-latest	768	137 Million	0.950066	0.954050

Code

prec_res = pd.concat(prec).unstack()
prec_res

			Train/Test	Test	Train
Collection	Vectoriser	Vectoriser Dimensions	Vectoriser Parameters
gemini-embedding-001	quran-embedding-prefect-gemini-embedding-001	768	Undisclosed	0.048495	0.163671
nomic-embed-text-v2-moe:latest	quran-embedding-prefect-nomic-embed-text-v2-moe-latest	768	475.29 Million	0.669019	0.717549
nomic-embed-text:latest	quran-embedding-prefect-nomic-embed-text-latest	768	137 Million	0.953977	0.957905

4.2 Word Embeddings vs Heuristics

- In terms of performance, the heuristic approach which is described fully in Section 6.1 achieved comparable (or better) recall and precision to the LLMs across all languages i.e. judging by the quantiles of distribution of recall and precision. It is however possible that a neural network trained on the embeddings would reveal a different result. It is also likely that a using a larger and more diverse dataset will show an entirely different pattern and this is a subject of further exploration. - The heuristic approach is also more informative about the languages than the LLMs. This is shown by the mutual information (MI) between the languages and the language predicted by the heuristic. - The detailed MI, recall, and precision obtained for each approach by each language is provided in Section 6.2

Code

agg_mi = [] 

agg_recall = []

agg_precision = [] 

count = 0

for c in collections: 
    samples[c]['heur_lang_prop'] = samples[c]['text'].apply(lambda e:detect_language(e))
    samples[c]['heur_lang'] = samples[c]['heur_lang_prop'].apply(lambda e:e['main_language'])
    samples[c]['heur_lang_pred_idx'] = pd.Series(tar_encoder[c].transform(samples[c]['heur_lang']))

    num_obs = samples[c].groupby(['language']).size()
    
    # mutual information
    mr = samples[c].groupby(['language']).apply(lambda df:compute_norm_mutual_info(df,'target_idx','heur_lang_pred_idx'))
    mr = pd.DataFrame(mr, columns = [f"Heuristic - MI"])
   
    mk = samples[c].groupby(['language']).apply(lambda df:compute_norm_mutual_info(df, 'target_idx', 'knn_predictions'))
    mk = pd.DataFrame(mk,columns=[f"UMAP+KNN - {c} - MI"])

    if count == 0:
        agg_mi.append(mr)
        
    
    agg_mi.append(mk)
    
    # recall 
    rr = samples[c].groupby(['language']).apply(lambda df:compute_recall(df,'target_idx', 'heur_lang_pred_idx'))
    rr = pd.DataFrame(rr, columns=[f"Heuristic - Recall"])
    
    rk = samples[c].groupby(['language']).apply(lambda df:compute_recall(df, 'target_idx', 'knn_predictions'))
    rk = pd.DataFrame(rk, columns=[f"UMAP+KNN - {c} - Recall"])

    if count ==0:
        agg_recall.append(rr)
    
    agg_recall.append(rk)
    
    # precision
    pr = samples[c].groupby(['language']).apply(lambda df:compute_precision(df, 'target_idx', 'heur_lang_pred_idx'))
    pr = pd.DataFrame(pr, columns=[f"Heuristic - Precision"])
    
    pk = samples[c].groupby(['language']).apply(lambda df:compute_precision(df, 'target_idx', 'knn_predictions'))
    pk = pd.DataFrame(pk, columns=[f"UMAP+KNN - {c} - Precision"])

    if count == 0: 
        agg_precision.append(pr)
    
    agg_precision.append(pk)

    count += 1

Code

agg_mi_res = pd.concat(agg_mi, axis=1)
agg_mi_desc = agg_mi_res.describe()
agg_mi_desc.index = ['Number of Languages','Avg. MI', "Std. MI", "Minimum MI", "25th Percentile", "50th Percentile", "75th Percentile", "Maximum MI" ]

agg_rec_res = pd.concat(agg_recall,axis=1)
agg_rec_desc = agg_rec_res.describe() 
agg_rec_desc.index = ['Number of Languages','Avg. Recall', "Std. Recall", "Minimum Recall", "25th Percentile", "50th Percentile", "75th Percentile", "Maximum Recall" ]

agg_prec_res = pd.concat(agg_precision, axis=1)
agg_prec_desc = agg_prec_res.describe()
agg_prec_desc.index = ['Number of Languages','Avg. Precision', "Std. Precision", "Minimum Precision", "25th Percentile", "50th Percentile", "75th Percentile", "Maximum Precision" ]

Mutual Information (MI) per Approach

Code

style_summary_table(agg_mi_desc)

	Heuristic - MI	UMAP+KNN - gemini-embedding-001 - MI	UMAP+KNN - nomic-embed-text:latest - MI	UMAP+KNN - nomic-embed-text-v2-moe:latest - MI
Number of Languages	21	21	21	21
Avg. MI	0.38	0.00	0.00	0.00
Std. MI	0.50	0.00	0.00	0.00
Minimum MI	0.00	0.00	0.00	0.00
25th Percentile	0.00	0.00	0.00	0.00
50th Percentile	0.00	0.00	0.00	0.00
75th Percentile	1.00	0.00	0.00	0.00
Maximum MI	1.00	0.00	0.00	0.00

Recall per Approach

Code

style_summary_table(agg_rec_desc)

	Heuristic - Recall	UMAP+KNN - gemini-embedding-001 - Recall	UMAP+KNN - nomic-embed-text:latest - Recall	UMAP+KNN - nomic-embed-text-v2-moe:latest - Recall
Number of Languages	21	21	21	21
Avg. Recall	0.85	0.12	0.95	0.70
Std. Recall	0.26	0.07	0.09	0.30
Minimum Recall	0.00	0.06	0.64	0.16
25th Percentile	0.88	0.07	0.97	0.40
50th Percentile	0.99	0.10	0.99	0.71
75th Percentile	1.00	0.13	1.00	0.98
Maximum Recall	1.00	0.37	1.00	1.00

Precision per Approach

Code

style_summary_table(agg_prec_desc)

	Heuristic - Precision	UMAP+KNN - gemini-embedding-001 - Precision	UMAP+KNN - nomic-embed-text:latest - Precision	UMAP+KNN - nomic-embed-text-v2-moe:latest - Precision
Number of Languages	21	21	21	21
Avg. Precision	0.50	0.05	0.18	0.08
Std. Precision	0.38	0.00	0.06	0.04
Minimum Precision	0.00	0.05	0.11	0.06
25th Percentile	0.20	0.05	0.14	0.06
50th Percentile	0.33	0.05	0.17	0.06
75th Percentile	1.00	0.05	0.20	0.09
Maximum Precision	1.00	0.05	0.33	0.20

5.0 Lessons Learned

Horses for Courses: In certain situations, the least sophisticated approach is the most suitable. For instance, the heuristic approach that uses regular expressions for language detection could be more suitable for detecting languages than LLM based word representations.

5.1 Next Steps

Although the performance of the heuristic is explainable i.e. it is largely based on the scripts used in the text, it is unclear whether this is the same for the LLMs.

6.0 Appendix

6.1 Heuristic Language Detection

    flowchart TD
    A["detect_language(e)"] --> B["NFC-normalise text"]
    B --> C["Count chars per script block<br/>(Latin, Arabic, Cyrillic,<br/>Ethiopic, Thaana, Tamil, Thai, CJK)"]
    C --> D{"Any script<br/>chars found?"}
    D -->|no| UND(["main_language = ''"]):::und

    D -->|"yes: take dominant script"| E{"Script type?"}

    %% ---- Stage 1: unique scripts decided outright ----
    E -->|Ethiopic| U1(["am"]):::win
    E -->|Thaana| U2(["dv"]):::win
    E -->|Tamil| U3(["ta"]):::win
    E -->|Thai| U4(["th"]):::win
    E -->|CJK| U5(["zh"]):::win

    %% ---- Stage 2: shared scripts need disambiguation ----
    E -->|Arabic| AR["Score on EXCLUSIVE markers only"]
    E -->|Cyrillic| CY["Score on EXCLUSIVE markers only"]
    E -->|Latin| LA["Score on exclusive letters<br/>+ digraphs + stopwords"]

    AR --> AR1{"priority check"}
    AR1 -->|"Urdu retroflex / ye-barree"| ARu(["ur"])
    AR1 -->|"Sindhi implosives"| ARs(["sd"])
    AR1 -->|"Persian p/ch/zh/g (no ur,sd)"| ARf(["fa"])
    AR1 -->|"else Arabic kaf/ye / fallback"| ARa(["ar"])

    CY --> CY1{"exclusive Cyrillic marks"}
    CY1 -->|"Tajik ii/uu/j"| CYt(["tg"])
    CY1 -->|"Uzbek o-breve"| CYu(["uz"])
    CY1 -->|"Serbian dje/lje/nje"| CYb(["bs"])
    CY1 -->|"else / fallback"| CYr(["ru"])

    LA --> LA1["Per-language score:<br/>char x3, digraph x1, stopword x2"]
    LA1 --> LA2{"argmax<br/>(en,es,nl,sq,tr,bs,ha,so,uz)"}
    LA2 --> LAout(["best Latin lang"])

    %% ---- Final threshold ----
    ARu & ARs & ARf & ARa & CYt & CYu & CYb & CYr & LAout --> T{"best score >=<br/>MIN_CONFIDENCE?"}
    T -->|yes| WIN(["main_language = best"]):::win
    T -->|no| UND

    classDef win fill:#dcfce7,stroke:#16a34a,color:#14532d;
    classDef und fill:#fee2e2,stroke:#dc2626,color:#7f1d1d;

Here’s what each node in the diagram represents:

A — detect_language(e): entry point; takes one text string.
B — NFC-normalise: Unicode-normalises the text so composed/decomposed forms of the same character count identically.
C — Count chars per script block: tallies how many characters fall in each script range (Latin, Arabic, Cyrillic, Ethiopic, Thaana, Tamil, Thai, CJK). This is the single measurement that drives routing.
D — Any script chars found?: guards against empty/numeric/punctuation-only input. If nothing matches, the string is unclassifiable → '' (empty string).
E — Script type?: picks the dominant script (highest count from C) and branches on it. This is the key move — decide the script once, then only consider languages that use it.
U1–U5 — am / dv / ta / th / zh: the five languages with an exclusive script (Ethiopic, Thaana, Tamil, Thai, CJK). Script alone identifies them, so they’re returned immediately with no further scoring.
AR / CY / LA — “score on exclusive markers”: the three shared-script buckets. Each scores candidate languages only on features that differ between them — never on the shared range that caused the original collisions.
AR1 (Arabic priority check) → ur / sd / fa / ar: checks distinctive letters in priority order — Urdu retroflexes/yeh-barree first, then Sindhi implosives, then Persian consonants (only if no Urdu/Sindhi marks), else Standard Arabic or fallback.
CY1 (Cyrillic marks) → tg / uz / bs / ru: routes on exclusive Cyrillic letters — Tajik (ӣ ӯ ҷ), Uzbek (ў), Serbian (ђ ј љ њ ћ џ) — defaulting to Russian when none appear.
LA1 — per-language score: for the nine Latin languages, adds up weighted evidence — exclusive letter ×3, digraph ×1, stopword ×2. Stopwords are what separate en/nl/so, which have no unique letters.
LA2 — argmax → best Latin lang: picks the highest-scoring Latin language.
T — best score ≥ MIN_CONFIDENCE?: final gate. Every shared-script branch funnels here; a winner that clears the threshold becomes main_language, otherwise the result is '' (empty string).
WIN / UND: the two terminal outcomes — a confident language code, or the unclassified bucket (main_language == '', the blank row in your recall table).

6.2 Metrics per Language

6.2.1 Mutual Information per Language

Code

style_dataframe_heatmap(agg_mi_res)

	Heuristic - MI	UMAP+KNN - gemini-embedding-001 - MI	UMAP+KNN - nomic-embed-text:latest - MI	UMAP+KNN - nomic-embed-text-v2-moe:latest - MI
language
	1.00	0.00	0.00	0.00
am	1.00	0.00	0.00	0.00
ar	1.00	0.00	0.00	0.00
bs	0.00	0.00	0.00	0.00
dv	1.00	0.00	0.00	0.00
en	0.00	0.00	0.00	0.00
es	0.00	0.00	0.00	0.00
fa	0.00	0.00	0.00	0.00
ha	0.00	0.00	0.00	0.00
nl	0.00	0.00	0.00	0.00
ru	1.00	0.00	0.00	0.00
sd	0.00	0.00	0.00	0.00
so	0.00	0.00	0.00	0.00
sq	0.00	0.00	0.00	0.00
ta	1.00	0.00	0.00	0.00
tg	0.00	0.00	0.00	0.00
th	1.00	0.00	0.00	0.00
tr	0.00	0.00	0.00	0.00
ur	0.00	0.00	0.00	0.00
uz	0.00	0.00	0.00	0.00
zh	1.00	0.00	0.00	0.00

6.2.2 Recall per Language

Code

style_dataframe_heatmap(agg_rec_res)

	Heuristic - Recall	UMAP+KNN - gemini-embedding-001 - Recall	UMAP+KNN - nomic-embed-text:latest - Recall	UMAP+KNN - nomic-embed-text-v2-moe:latest - Recall
language
	0.00	0.16	0.88	1.00
am	1.00	0.14	0.64	0.98
ar	1.00	0.37	1.00	0.92
bs	0.95	0.12	0.98	0.35
dv	1.00	0.12	0.96	0.99
en	0.91	0.13	1.00	0.29
es	0.88	0.10	0.99	0.37
fa	1.00	0.09	1.00	0.40
ha	0.61	0.10	0.99	0.99
nl	0.94	0.11	1.00	0.54
ru	1.00	0.07	1.00	0.58
sd	0.35	0.07	0.98	0.98
so	0.99	0.09	0.99	1.00
sq	0.99	0.07	1.00	0.25
ta	1.00	0.07	0.95	0.71
tg	0.70	0.06	0.97	0.99
th	1.00	0.08	0.73	0.66
tr	0.97	0.06	0.99	0.16
ur	1.00	0.21	1.00	0.61
uz	0.60	0.07	0.98	0.98
zh	1.00	0.21	0.99	0.87

6.2.3 Precision per Language

Code

style_dataframe_heatmap(agg_prec_res)

	Heuristic - Precision	UMAP+KNN - gemini-embedding-001 - Precision	UMAP+KNN - nomic-embed-text:latest - Precision	UMAP+KNN - nomic-embed-text-v2-moe:latest - Precision
language
	0.00	0.05	0.25	0.20
am	1.00	0.05	0.17	0.06
ar	1.00	0.05	0.33	0.06
bs	0.20	0.05	0.20	0.06
dv	1.00	0.05	0.12	0.17
en	0.17	0.05	0.25	0.06
es	0.20	0.05	0.17	0.06
fa	0.50	0.05	0.20	0.07
ha	0.12	0.05	0.12	0.14
nl	0.20	0.05	0.25	0.06
ru	1.00	0.05	0.25	0.07
sd	0.25	0.05	0.14	0.06
so	0.20	0.05	0.17	0.17
sq	0.20	0.05	0.17	0.06
ta	1.00	0.05	0.12	0.06
tg	0.50	0.05	0.17	0.10
th	1.00	0.05	0.12	0.07
tr	0.17	0.05	0.11	0.06
ur	0.33	0.05	0.17	0.06
uz	0.50	0.05	0.14	0.09
zh	1.00	0.05	0.17	0.07